[Logs+] Change default of ignore_malformed to true in logs-*-* data streams#95329
Merged
felixbarny merged 9 commits intoelastic:mainfrom Apr 27, 2023
Merged
Conversation
elasticsearchmachine
added
needs:triage
eyalkoren
changed the title
ignore_malformed to true in logs-*-* data streams
felixbarny
reviewed
Apr 18, 2023
ruflin
reviewed
Apr 18, 2023
felixbarny
added
the
:StorageEngine/Data streams
elasticsearchmachine
added
Team:Data Management (obsolete)
Collaborator
|
Pinging @elastic/es-data-management (Team:Data Management) |
felixbarny
approved these changes
Apr 18, 2023
...s/src/yamlRestTest/resources/rest-api-spec/test/data_stream/200_logs_datastream_defaults.yml
Outdated
Show resolved
Hide resolved
...s/src/yamlRestTest/resources/rest-api-spec/test/data_stream/200_logs_datastream_defaults.yml
Outdated
Show resolved
Hide resolved
...s/src/yamlRestTest/resources/rest-api-spec/test/data_stream/200_logs_datastream_defaults.yml
Show resolved
Hide resolved
felixbarny
reviewed
Apr 18, 2023
...s/src/yamlRestTest/resources/rest-api-spec/test/data_stream/200_logs_datastream_defaults.yml
Outdated
Show resolved
Hide resolved
Collaborator
|
Hi @eyalkoren, I've created a changelog YAML for you. |
Collaborator
|
Hi @eyalkoren, I've updated the changelog YAML for you. |
ruflin
reviewed
Apr 19, 2023
Contributor
ruflin
left a comment
ruflin
left a comment
There was a problem hiding this comment.
@felixbarny I do not want to expand the scope of this PR here. But can we keep track somewhere of these changes because I think most of these we should eventually also apply to metrics, traces, etc. for all of the data stream naming scheme.
…ormed' into logs-data-streams-ignore_malformed
Member
|
Let's get in #95469 before merging this to make it easier for users to opt-out, without having to duplicate the whole default logs template. |
Contributor
Author
|
This issue is currently blocked on #95481 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes #95224
As part of our effort to accept all logs by default, one of the first issues we want to address is the rejection of whole log event documents due to field type not matching the corresponding mapping.
The intention of this issue is to change the default of
ignore_malformedtotruespecifically forlogs-*-*data streams, so that the log event will be indexed and the incorrectly typed field will be ignored. The ignored fields, as well as their values, are available through query, as shown in the added test.Checklist
ignore_malformed: trueat index level forlogs-*-*data streams@timestampfield, for which it should remainignore_malformed: false